Mapping the Reliability-Readability Gap in the Education of Patients With Age-Related Macular Degeneration Across 6 Large Language Models: Comparative Evaluation Study

Introduction

Age-related macular degeneration (AMD) is a leading cause of irreversible vision loss in older adults worldwide [1]. Management of this condition, which encompasses intravitreal anti–vascular endothelial growth factor (VEGF) therapy, nutritional supplementation, and lifestyle modification, is complex and requires the informed and active participation of the patient [2]. Effective patient education is therefore crucial, yet often constrained by limited clinical time, driving patients to seek information online [3]. Online patient education materials in ophthalmology have been shown to vary substantially in readability, accountability, and suitability, and many exceed recommended reading levels. Similar concerns have also been reported for artificial intelligence (AI)–generated ophthalmology patient education materials [3-5]. These findings support the need to evaluate whether large language model (LLM)–generated AMD information is both reliable and understandable for patients.

The rapid emergence of LLMs as conversational agents presents a transformative opportunity for scalable health communication. Tools such as OpenAI’s ChatGPT are accessed by hundreds of millions of users and demonstrate an ability to simplify medical text toward lower reading levels upon explicit instruction [6,7]. In ophthalmology, early evaluations suggest that LLMs can generate generally coherent responses to common patient queries about conditions such as AMD and glaucoma, aligning with their “impressive but mixed” performance across broader medical contexts [3,8].

However, 2 critical and interrelated barriers impede their direct clinical deployment for patient education. First, readability remains persistently excessive and should be interpreted within the broader framework of health literacy. Health literacy determines whether patients can obtain, understand, evaluate, and use medical information in decisions about their care. The National Institutes of Health, the US Department of Health and Human Services, and the American Medical Association have recommended that patient education materials be written at approximately a sixth-grade reading level [9]. This issue is particularly relevant to AMD, which predominantly affects older adults and may impose additional reading burdens because of central visual impairment. Nevertheless, most ophthalmic education content, including traditional handouts and website-based materials, far exceeds this recommendation [4,10,11]. This concern also extends to AMD-specific patient education materials and AI-generated ophthalmology patient education materials, which frequently remain above recommended reading levels [4,5]. Whether improved readability can be achieved without compromising informational reliability remains an empirical question in the education of patients with AMD. Therefore, this study jointly evaluated readability and reliability rather than treating readability as an isolated linguistic outcome.

Current evaluations in ophthalmology, while noting issues of factual omission or nuance, have not systematically quantified this trade-off using standardized, multidimensional reliability instruments (eg, DISCERN and Ensuring Quality Information for Patients [EQIP]) alongside rigorous readability metrics [4,12,13].

Crucially, the performance landscape of state-of-the-art LLMs remains unmapped in specialized clinical domains such as AMD. Existing multimodel comparisons are either generic or limited in scope, leaving a significant gap in understanding how newer, advanced models (eg, Grok 4, Claude-Sonnet 4.5, and DeepSeek-v3.2) perform relative to each other [14]. Furthermore, assessments conducted under a “zero-shot”– prompting scenario, which best simulates a naive patient’s initial and unoptimized query, are essential to establish a realistic performance baseline but are seldom the focus [15,16].

Therefore, to move from generic caution toward actionable evidence, this study aims to (1) conduct a systematic, head-to-head evaluation of 6 leading LLMs on a curated set of AMD patient–facing prompts; (2) simultaneously quantify output quality using 4 established reliability tools (DISCERN, EQIP, Global Quality Scale [GQS], and Journal of the American Medical Association [JAMA] benchmark criteria) and 6 readability formulas; and (3) explicitly analyze the interplay—or trade-off—between reliability and readability under a minimalist, real-world-query simulation. By establishing this comprehensive performance atlas, we provide the empirical foundation necessary for the informed selection, targeted optimization, and safe, human-supervised integration of LLMs into ophthalmic patient care.

Methods

Prompt Development and Selection

A set of 30 English-language prompts on AMD was formulated through a multistep process (Textbox 1). First, an analysis of global Google Trends data from October 10, 2020, to October 10, 2025, was conducted based on the MeSH term “Age-Related Macular Degeneration” to identify frequently used patient search queries. This yielded 25 high-frequency search phrases. Second, 8 clinically focused prompts were extracted from the “Evidence-Based Guidelines for Diagnosis and Treatment of Age-Related Macular Degeneration in China (2023)” [17]. For these Chinese guideline–derived prompts, the initial English translation was performed by ZL immediately after extraction. The translated prompts were then reviewed and validated by XM for semantic equivalence, ophthalmic terminology, and consistency with English clinical expression. Subsequently, 10 topic prompts were extracted from the 2025 American Academy of Ophthalmology “Age-Related Macular Degeneration Preferred Practice Pattern” [18]. Finally, from these 43 initial prompts, irrelevant or duplicate entries were removed, resulting in 30 unique English-language prompts (Textbox 1). These finalized English-language prompts were entered verbatim into the 6 evaluated LLMs without modification, and the responses were collected for further analysis. The final sample size of 30 prompts was selected a priori to provide a representative cross-section of common public-search, guideline-derived, and clinically relevant AMD information needs while preserving the feasibility of manual-blinded evaluation by ophthalmologist raters.

LLMs and Query Process

Six publicly available, general-purpose conversational LLMs were evaluated: ChatGPT-5.1-auto (OpenAI; August 7, 2025), DeepSeek-V3.2-Exp, displayed in the public interface as DeepSeek-v3.2 (DeepSeek; September 25, 2025), Gemini-2.5-Flash-Thinking (Google; June 17, 2025), Grok 4 (xAI; July 9, 2025), Claude Sonnet 4.5 (Anthropic; September 29, 2025), and Qwen3-Max (Alibaba Cloud; September 24, 2025). These dates were used to document the public model versions or modes available during the data collection period and do not imply access to fixed back-end model-weight snapshots. These models were selected because they were broadly accessible during the study period, represented major international and Chinese AI developers, were available as identifiable model versions or modes, and could be evaluated under the same zero-shot, prompt-only procedure. Dedicated retrieval-augmented answer engines designed primarily around live search and source ranking, such as Perplexity, were not included. However, this exclusion did not mean that all evaluated public web interfaces functioned as pure stand-alone base models. The study evaluated each model as encountered by a typical public user under default interface settings. During review of the raw outputs, ChatGPT-5.1-auto and Qwen3-Max displayed retrieval- or citation-related artifacts in at least some default interface responses, including source links or Urchin Tracking Module (UTM)–tracked links and bracketed web citations. These features were not manually activated, disabled, standardized, or selectively removed. Therefore, the results should be interpreted as a comparison of default public interface responses rather than a controlled comparison of nonretrieval stand-alone model capabilities. All queries were conducted between October 10 and November 25, 2025, through the official publicly accessible web interfaces of the 6 evaluated models. No API calls, automated scripts, browser extensions, plug-ins, or programmatic access methods were used. Each finalized English-language prompt was manually entered by the authors into each model interface, and the corresponding response was manually extracted and recorded for analysis. All Google Trends searches and LLM queries were conducted in private or incognito browsing mode after signing out of Google and other personal accounts. Browser cache and cookies were cleared before each session to minimize local browser session effects. All prompts and model outputs included in the analysis were in English. Each prompt was entered in a separate new session, without prior conversational history, additional context, custom instructions, user-specified system prompts, repeated prompt optimization, regeneration, or selective response replacement. Default public user interface settings were used. Accordingly, API-level generation parameters, including temperature, top-p, maximum token length, and frequency or presence penalties, were not displayed to or controllable by the investigators and were not manually set. A single response from each model for each prompt was collected for analysis.

Response Preprocessing and Blinding

Before scoring, all model responses were transferred from the original model-specific files into a standardized scoring template and assigned anonymized response codes. Model names, file names, interface labels, logos, time stamps, ordering information, and any explicit self-identifying boilerplate text indicating the model name or developer were manually removed or redacted before rater review. This redaction was limited strictly to model-identifying information and did not alter any substantive clinical content, wording, recommendations, references, uncertainty statements, or safety-related statements in the responses. However, URL-level tracking parameters embedded within reference links, including UTM parameters, were not systematically scrubbed before scoring because references were otherwise preserved to avoid altering attribution-related content. Therefore, although explicit model names, interface labels, and self-identifying boilerplate were removed, some reference URLs may have retained model-identifying tracking strings. The 2 ophthalmologist raters received only anonymized prompt response pairs and were not informed of the model source. When discrepancies occurred, a third ophthalmologist (CM) adjudicated the score using the same anonymized materials.

Outcome Measures

Reliability Assessment

Two board-certified ophthalmologists (ZL and X.M) with over 10 years of clinical experience independently evaluated the anonymized and preprocessed responses using 4 established tools while remaining blinded to the model sources. In case of disagreement between the 2 raters, a third ophthalmologist of equivalent qualification served as the final adjudicator.

DISCERN

DISCERN was used to assess the quality of information regarding “treatment choices” in patient education materials [19]. It consists of 16 items divided into 3 sections: reliability of information (items 1‐8), specific quality of information on treatment choices (items 9‐15), and an overall rating (item 16). Each item is scored from 1 to 5, with a total score range of 16‐80. Higher scores indicate better overall quality. For this study, scores were defined as follows: 16‐26: “very poor,” 27‐38: “poor,” 39‐50: “fair,” 51‐62: “good,” and 63‐80: “excellent.” [20]

Ensuring Quality Information for Patients

EQIP was used to assess the overall quality of health education materials regarding information sources, accuracy, comprehensibility, and presentation format [21]. It consists of 20 items. Each item is answered “Yes/No,” with “Yes” scoring 1 point and “No” scoring 0 points. The final score is the percentage of “Yes” items out of the total (ie, proportion of “Yes” × 100). Higher scores indicate better material quality [22].

Global Quality Scale

GQS is a single-item, 5-point Likert scale for subjective overall evaluation of health education materials: 1 point for “Very poor,” 2 for “Poor,” 3 for “Fair,” 4 for “Good,” and 5 for “Excellent.”

JAMA Benchmark Criteria

JAMA criteria focus on the standardization and transparency of online health information, including 4 items: authorship, attribution of sources and references, disclosure of conflicts of interest (or sponsorship or advertising), and currency. Each item is scored “Yes/No,” with “Yes” scoring 1 point and “No” scoring 0 points, for a total score of 0‐4. Higher scores indicate better performance in information transparency and standardization [23]. In this study, the JAMA benchmark was applied uniformly across all model outputs as a structured proxy for visible transparency and verifiability features within the response text, rather than as a direct measure of factual accuracy, clinical safety, true human authorship, or actual financial conflicts of interest. Because citation display may be an interface-level feature rather than an intrinsic property of the underlying model, JAMA benchmark scores in this study should be interpreted as visible transparency under default public interface conditions.

Hallucination Assessment Scope

For transparency, hallucination was operationally defined as fabricated clinical facts, guideline-inconsistent recommendations, unsupported certainty regarding diagnosis or treatment, or nonverifiable references presented as evidence. However, hallucination was not prespecified as a stand-alone analytic end point in this study. During clinical evaluation, raters assessed the informational reliability, transparency indicators, overall quality, and readability of model responses using DISCERN, EQIP, GQS, JAMA benchmark criteria, and readability formulas. Although unsupported or misleading content could influence these quality scores, the raters did not perform claim-level fact checking, assign binary hallucination labels, or record model-level hallucination frequencies. Therefore, hallucination frequency was not reported as an independent outcome.

Readability Assessment

Because all prompts and model outputs analyzed in this study were in English, English-language readability formulas were used. No Chinese-language prompts, untranslated Chinese-language prompts, or Chinese-language model outputs were included in the readability assessment.

Responses were analyzed using six standard readability formulas calculated via an online tool (readabilityformulas.com):

1. Automated Readability Index (ARI) [24]:

${\displaystyle {\displaystyle 4.71\left(\frac{\text{Characters}}{\text{Words}}\right)+0.5\left(\frac{\text{Words}}{\text{Sentences}}\right)-21.43}}$

2. Flesch Reading Ease Score (FRES) [25]:

${\displaystyle {\displaystyle 206.835-1.015\left(\frac{\text{Words}}{\text{Sentences}}\right)-84.6\left(\frac{\text{Syllables}}{\text{Words}}\right)}}$

3. Gunning Fog Index (GFI) [25]:

${\displaystyle {\displaystyle 0.4\left[\left(\frac{\text{Words}}{\text{Sentences}}\right)+100\left(\frac{\text{Complex words}}{\text{Words}}\right)\right]}}$

4. Flesch-Kincaid Grade Level (FKGL) [25]:

${\displaystyle {\displaystyle 0.39\left(\frac{\text{Words}}{\text{Sentences}}\right)+11.8\left(\frac{\text{Syllables}}{\text{Words}}\right)-15.59}}$

5. Coleman-Liau Index (CL) [25]:

${\displaystyle {\displaystyle \begin{array}{rl}& 5.89\left(\frac{\text{Characters}}{\text{Words}}\right)\\ & -0.3\left(\frac{\text{Sentences}}{\text{Words}}\right)-15.8\end{array}}}$

6. Simple Measure of Gobbledygook (SMOG) [25]:

${\displaystyle {\displaystyle 14.430\times \sqrt{\text{Polysyllables }\times \frac{30}{sentences}}+3.1291}}$

The readability scores were compared and analyzed with the sixth-grade level of readability recommended by the American Medical Association and the National Institutes of Health [26]. This threshold was considered appropriate for the education of patients with AMD because it has been used in prior studies of AMD-specific and geriatric patient education materials, and because AMD predominantly affects older adults, for whom central visual impairment may further increase the practical burden of reading and processing complex written health information [5,27]. Following prior readability studies, including the referenced PeerJ article, the accepted readability level was defined as ≥80.0 for the FRES and ≤6 for the other 5 grade-level formulas. For interpretation, lower FRES values indicate more difficult reading, whereas higher ARI, GFI, FKGL, CL, and SMOG values indicate greater reading difficulty; therefore, FRES values below 80.0 or grade-level formula scores above 6 were interpreted as exceeding the recommended sixth-grade readability level.

Statistical Analysis

Overview

All statistical analyses were performed using IBM SPSS Statistics (version 29.0; IBM Corp) and R (version 4.3.2; R Core Team). First, the mean and standard deviation for each of the 6 models across all 30 prompts were calculated. Interrater agreement between the 2 independent assessors on the 4 reliability measures (DISCERN, EQIP, GQS, and JAMA benchmark criteria) was calculated using Cohen κ. Cohen κ values and corresponding 95% CIs were calculated from the paired preadjudication ratings of the 2 independent assessors. Agreement was categorized as follows: κ≤0, no agreement; 0.01‐0.20, slight; 0.21‐0.40, fair; 0.41‐0.60, moderate; 0.61‐0.80, substantial; and 0.81‐1.00, almost perfect agreement.

Reliability Analysis

Cohen κ was used to assess interrater agreement for the 4 reliability tools. Descriptive statistics (mean, SD) were calculated for each model across all 30 prompts for each measure. The Friedman test was used to examine overall differences among the 6 models for each measure, followed by post hoc pairwise comparisons with Holm-Bonferroni adjustment for multiple comparisons when significant. A P value of <.05 was set as the threshold for statistical significance. Composite reliability and readability z scores were calculated as described in the Statistical Analysis subsection and were used to generate the composite visualization.

Readability Analysis

Readability scores were calculated for each model’s responses using the established readability metrics. Given the repeated-measures design, in which each of the 30 standardized AMD prompts was answered by all 6 models, the Friedman test was used to compare differences in readability scores among the 6 models. When the overall Friedman test was significant, post hoc pairwise comparisons were performed with Holm-Bonferroni adjustment. To assess whether the generated text met the sixth-grade reading level, the Wilcoxon signed-rank test was used to compare readability scores against the standardized sixth-grade threshold. P values were calculated for all tests, with P<.05 considered statistically significant.

Regarding Figure 1, average scores for each model were normalized by indicator using minimum-maximum normalization, resulting in scores ranging from 0 to 1.

‎

For “higher-is-better” indicators, the following equation was used:

${\displaystyle {\displaystyle S_{m,k}=\frac{{\overline{x}}_{m,k}-\underset{k}{min}}{\underset{k}{max}-\underset{k}{min}}}}$

For “lower-is-better” readability grade-level indicators, the following equation was used:

${\displaystyle {\displaystyle S_{m,k}=\frac{\underset{k}{max}-{\overline{x}}_{m,k}}{\underset{k}{max}-\underset{k}{min}}}}$

Regarding Figure 2, all 180 observations were Z-standardized:

${\displaystyle {\displaystyle Z_{i,k}=\frac{x_{i,k}-{\mu }_k}{{\sigma }_k}}}$

For “lower-is-better” readability grade-level indicators, the sign was inverted:

${\displaystyle {\displaystyle Z_{i,k}^{\ast }=-Z_{i,k}}}$

Composite reliability and readability were calculated as unweighted averages:

${\displaystyle {\displaystyle {\displaystyle R_i=\frac{1}{4}\sum Z_{i,k}^{\ast },Q_i=\frac{1}{6}\sum Z_{i,k}^{\ast }}}}$

‎

Regarding Figure 3, the Friedman test was used to compare overall differences among the 6 models for each indicator:

${\displaystyle {\displaystyle {\displaystyle X_F^2=\frac{12}{nk(k+1)}\sum R_j^2-3n(k+1)}}}$

‎

If significant, pairwise comparisons were conducted. The standardized test statistic was:

${\displaystyle {\displaystyle Z=\sqrt{\frac{\left|\overline{r_a}-\overline{r_b}\right|}{\frac{k(k+1)}{6n}}}}}$

The Holm-Bonferroni correction was applied. The heatmap displays $-{\log }_{\text{10}}\text{(}{\text{p}}_{\text{Holm}}\text{)}$.

Ethical Considerations

This study did not involve human participants, direct interaction with individuals, identifiable private information, or individual-level patient data. The analysis was based on publicly available search-trend information, published clinical guidelines and recommendations, and textual responses generated by publicly accessible LLMs. Therefore, formal ethics review was not required, informed consent was not applicable, and no ethics approval number was assigned. No personal identifiers, private user information, or patient records were collected or analyzed. No compensation was provided because no participants were recruited. The manuscript and supplementary materials do not contain images, screenshots, or other content that could identify individual participants or users.

Results

Overview

A total of 180 responses (30 prompts × 6 models) were analyzed. Interrater agreement for all 4 reliability instruments was substantial to almost perfect, with κ values of 0.72 for DISCERN (95% CI 0.635‐0.802), 0.80 for EQIP (95% CI 0.717‐0.875), 0.92 for GQS (95% CI 0.864‐0.959), and 0.97 for JAMA benchmark criteria (95% CI 0.942‐0.988). These findings indicated substantial to almost perfect preadjudication agreement between the 2 independent ophthalmologist assessors (ZL and XM); all scoring discrepancies were subsequently resolved by a third ophthalmologist (CM) of equivalent qualification, who served as the final adjudicator using the same anonymized materials.

Reliability Scores

Substantial between-model variation was observed across all reliability-related metrics. As shown in Table 1, Grok 4 achieved the highest mean DISCERN score (mean 46.40, SD 7.43), followed by DeepSeek-v3.2 (mean 43.33, SD 8.20). A similar pattern was observed for EQIP, where Grok 4 (mean 74.33, SD 9.07) and DeepSeek-v3.2 (mean 71.50, SD 9.11) outperformed the remaining models. For the JAMA benchmark criteria, Grok 4 generated the highest mean score (mean 1.97, SD 0.67), but Qwen3-Max showed a similarly high JAMA score (mean 1.90, SD 0.40), indicating comparatively strong visible transparency despite lower scores on DISCERN, GQS, and EQIP. Claude-Sonnet 4.5 showed the lowest overall reliability-related profile across the evaluated metrics, whereas Qwen3-Max showed lower non-JAMA reliability scores but relatively strong JAMA benchmark performance. These distributional differences are illustrated by the violin plots in Figure 4A-D, which present DISCERN, EQIP, GQS, and JAMA benchmark score distributions, respectively.

‎

Readability Scores

Readability assessments demonstrated that all models produced text considerably exceeding the recommended sixth-grade reading level (Table 2). This interpretation was based on the predefined cutoffs: all mean FRES values were below the accepted threshold of 80.0, and all mean grade-level formula scores were above 6, indicating text harder than the recommended sixth-grade level.

DeepSeek-v3.2 showed the most favorable overall readability profile, yielding the highest FRES score (mean 48.23, SD 9.16) and the lowest scores for ARI (mean 11.90, SD 2.14), GFI (mean 12.26, SD 1.81), FKGL (mean 9.95, SD 1.87), and CL (mean 11.91, SD 1.75). However, Claude-Sonnet 4.5 achieved the lowest SMOG score (mean 9.40, SD 1.59), followed by DeepSeek-v3.2 (mean 9.98, SD 1.50). In contrast, Gemini-2.5 Flash-Thinking, Grok 4, and Qwen3-Max produced text requiring substantially higher reading proficiency, with Qwen3-Max reaching the highest FKGL (mean 15.37, SD 2.85) and SMOG (mean 13.57, SD 2.49) levels. Violin plots in Figure 5 further demonstrate clear intermodel divergence in readability distributions. Wilcoxon signed-rank tests comparing readability scores with the prespecified sixth-grade thresholds confirmed that none of the 6 models met the recommended readability level. For ARI, GFI, FKGL, CL, and SMOG, values were significantly above the threshold of 6.0 across all models, whereas FRES values were significantly below the threshold of 80.0.

‎

Integrated Performance Visualization

After standardization, the cross-metric performance heatmap (Figure 1) revealed distinct model profiles. Grok 4 performed strongly across reliability-related metrics, particularly DISCERN, EQIP, and JAMA benchmark criteria, but this reliability advantage was not accompanied by favorable readability. Its responses showed mixed and generally less favorable readability than those of DeepSeek-v3.2. DeepSeek-v3.2 showed the most balanced performance profile, combining comparatively strong reliability with the most favorable overall readability. Consistent with Table 2, the composite z score scatterplots (Figure 2) showed that Grok 4 responses were distributed toward the high-reliability side but extended substantially into lower-readability regions, whereas DeepSeek-v3.2 occupied the more favorable reliability-readability region. Other models showed less favorable profiles across either reliability or readability dimensions.

Overall Between-Model Differences

Friedman tests confirmed significant between-model differences across all reliability and readability metrics (χ²₅ range, 43.859‐93.887; Kendall W range, 0.292‐0.626; all P<.001) (Table 3). Reliability indices showed the strongest effect sizes, particularly for GQS (χ²₅=93.887; W=0.626) and EQIP (χ²₅=82.132; W=0.548). Readability metrics also demonstrated robust intermodel variability, with the largest differences seen in SMOG (χ²₅=73.105; W=0.487) and FKGL (χ²₅=70.367; W=0.469).

Pairwise Comparisons

Holm-adjusted post hoc pairwise comparisons for reliability metrics are reported in Table 4, whereas corresponding pairwise comparisons for readability metrics are reported in Table 5. For reliability metrics, Grok 4 significantly outperformed Claude-Sonnet 4.5 across all 4 reliability-related metrics and outperformed Qwen3-Max on DISCERN, GQS, and EQIP but not on JAMA benchmark criteria. DeepSeek-v3.2 also showed significantly higher EQIP scores than several comparator models. For readability metrics, DeepSeek-v3.2 generated significantly easier-to-read responses than Gemini-2.5 Flash-Thinking, Grok 4, and Qwen3-Max across multiple readability indices, including ARI, FRES, FKGL, CL, and SMOG. These pairwise differences are visualized in Figure 3A for reliability metrics and Figure 3B for readability metrics. The corresponding Wilcoxon signed-rank statistics and exact 2-sided P values are reported in Table 6.

Discussion

Principal Findings

In this comparative evaluation of 6 publicly accessible LLMs for the education of patients with AMD, we found substantial model-dependent differences in informational reliability, transparency indicators, overall quality, and readability. No model met the recommended sixth-grade readability benchmark under zero-shot, single-turn prompting conditions [26]. These findings suggest a persistent reliability-readability gap in current LLM-generated education materials of patients with AMD. This gap is clinically relevant because the education of patients with AMD requires information that is not only reliable and transparent but also understandable for older adults who may face visual and health literacy barriers.

Comparison With Prior Work

Building on prior studies reporting that AI-generated patient education materials often remain above-recommended readability levels despite apparent coherence and informativeness, our findings show that higher informational reliability does not necessarily correspond to better readability [4,28]. This pattern is substantiated by the distinct model profiles shown in Figures 1, 4A-D, and 5A-F, and Tables 1 and 2. This divergence appears rooted in underlying design philosophies. Grok 4, which achieved the highest mean scores in DISCERN (mean 46.40, SD 7.43) and EQIP (mean 74.33, SD 9.07) (Table 1), concurrently produced text with among the highest grade–level indices (eg, ARI: mean 14.89, SD 3.38) (Table 2). In Figure 2, Grok 4 was positioned toward the high-reliability side but was also dispersed into lower-readability regions, consistent with a “precise expert” archetype: informationally robust yet linguistically dense. In contrast, DeepSeek-v3.2 occupied a unique and practically significant niche. It attained near-top-tier reliability while consistently generating the most readable text (highest FRES: mean 48.23, SD 9.16; lowest FKGL: mean 9.95, SD 1.87) (Table 2), presenting a more balanced profile in the Figure 1 heatmap. This observation serves as a critical proof-of-concept: the reliability-readability trade-off is not an immutable law but rather a mitigable and measurable engineering challenge. The profound and consistent intermodel differences, confirmed by highly significant Friedman tests (Table 3; all P<.001) and detailed in the pairwise comparison heatmap (Figure 3), underscore that these performance characteristics are inherent to model architecture and training, not artifacts of query phrasing. The persistence of these performance gaps under our “zero-shot” prompting protocol is a key methodological insight. By forgoing system prompts or complex instructions to mimic a naive user’s interaction, as described in the LLMs and Query Process subsection of the Methods, we reveal the “out-of-the-box” capability landscape that patients actually face. The failure of all models to approach the readability target under these conditions indicates that intrinsic model propensity, not superficial user instruction, is the primary governor of output complexity. This finding challenges the notion that simple prompt engineering alone will suffice for clinical adoption and shifts the onus for improvement decisively onto model developers and tailored fine-tuning protocols. This reframes readability not as a downstream formatting issue but as a core design constraint that must be addressed at the model or fine-tuning level.

Clinical Implications and Future Directions

These findings have practical implications for clinician-supervised use of LLMs in the education of patients with AMD. Patients’ understanding of AMD causes, pathophysiology, chronic progression, treatment options, and follow-up requirements can influence treatment expectations, adherence, timely care-seeking, and shared decision-making. Because LLM outputs may increasingly shape patients’ knowledge before or between clinical visits, their reliability and readability have practical implications for the education of patients with AMD. However, incomplete, overly complex, or unreliable explanations may mislead patients or delay appropriate care. Therefore, evaluating the reliability and readability of AI-generated AMD education is directly relevant to treatment-related decision-making outcomes. Direct, unsupervised provision of current LLM outputs to patients with AMD risks disseminating information that is inaccurate, incomprehensible, or both. Therefore, a pragmatic, 3-stage pathway for translation is warranted. In the context of AMD, such misalignment may concretely translate into misunderstandings regarding anti-VEGF treatment expectations, disease chronicity, or the distinction between neovascular and atrophic pathways.

The initial step involves evidence-based model selection for clinical settings exploring AI-drafted materials. Our data provide clear guidance: if the priority is reliability for subsequent expert editing, Grok 4 may be a suitable starting point. If the goal is a better-balanced starting point for communication, DeepSeek-v3.2 may be considered, although human simplification and clinical review would still be required [29].

Selection alone, however, is insufficient. Future research should move from passive benchmarking toward active intervention. Our study design provides the ideal framework for this: the curated 30-prompt AMD set (Textbox 1) and the comprehensive baseline performance data (Tables 1 and 2) can be used to rigorously test specific interventions. These include advanced prompt engineering strategies, such as chain-of-thought prompting for simplification, or fine-tuning protocols explicitly designed to close the quantified “readability gap,” for instance, by reducing the average FKGL of DeepSeek-v3.2 from 9.95 toward the target of 6, while vigilantly preserving reliability.

Ultimately, the most viable and safest near-term application is a synergistic human-AI workflow [30]. In this model, LLMs such as DeepSeek-v3.2 or Grok 4 act as powerful first-draft engines, significantly reducing clinician time spent on initial content creation [31]. This draft must then undergo mandatory review and literacy-sensitive simplification by a health care professional. This integrated workflow leverages AI’s scalability for efficiency while retaining irreplaceable human clinical judgment for safety, accuracy, and ultimate patient comprehension.

Strengths and Limitations

Strengths

This study has several strengths. First, it provides a head-to-head comparison of 6 publicly accessible state-of-the-art conversational LLMs within a clinically specific ophthalmic context, rather than evaluating a single model or a general medical prompt set. Second, the 30-prompt benchmark was developed from both real-world patient search behavior and authoritative AMD clinical guidance, thereby improving the clinical relevance and patient-facing applicability of the evaluation. Third, the study jointly assessed informational reliability and linguistic readability using 4 established quality instruments and 6 readability formulas, allowing the reliability-readability gap to be quantified rather than described qualitatively. Fourth, all responses were independently evaluated by 2 senior ophthalmologists blinded to model identity, with adjudication of disagreements, strengthening the rigor and reproducibility of the scoring process. Finally, the zero-shot design reflects the initial, unoptimized interaction that a naïve patient may have with a publicly available LLM, providing a realistic baseline for future model selection, prompt optimization, and clinician-supervised deployment.

Limitations

This study has several limitations. First, this study evaluated 6 selected public web interface LLM responses and used a limited set of 30 AMD-related prompts. Therefore, the findings should not be generalized to all AMD-related patient information needs, retrieval-augmented answer engines, locally deployed open-source models, API-only systems, or medical-specialized models. This limitation was partly mitigated by selecting broadly accessible models from major international and Chinese developers and by deriving the prompt set from both real-world search behavior and authoritative AMD guidance.

Second, only 1 response was collected for each model-prompt pair under a zero-shot, single-turn design. Although this approach was intended to reflect the first answer a naïve patient might receive, it does not capture within-model stochastic variability, repeated regenerations, or multiturn patient-LLM interactions. Public web interfaces may also include hidden system prompts, interface-level settings, dynamic context handling, dynamic safety filters, and back-end updates that were not controllable by the investigators. Therefore, the findings should be interpreted as version- and interface-dependent snapshots rather than fixed estimates of model performance. Future studies should incorporate repeated regenerations, API-based controlled sampling where available, and multiturn dialogue designs to better quantify response variability, ecological validity, and reproducibility. An additional limitation concerns default interface-level retrieval behavior. Although dedicated retrieval-augmented answer engines were excluded, several raw outputs indicated that some evaluated public interfaces, particularly ChatGPT-5.1-auto and Qwen3-Max, displayed source links, tracking links, or bracketed web citations under default settings. This weakens any interpretation of the study as a comparison of strictly stand-alone model knowledge. Such interface-level retrieval or citation behavior may have influenced response breadth, apparent currency, attribution, and JAMA benchmark scores. Accordingly, our findings should be interpreted as default public interface performance snapshots rather than fixed estimates of base-model capability. Future studies should separately evaluate browsing-enabled and browsing-disabled modes, where available, and explicitly record whether web retrieval or citation display was enabled, reporting retrieval or citation display status for each model and, ideally, for each response. Although explicit model names, interface labels, and self-identifying boilerplate were removed before scoring, complete blinding cannot be guaranteed. Some reference URLs retained in the scoring materials may have contained model-identifying tracking parameters, such as UTM tags, which could have compromised complete blinding for responses from models whose interfaces generated such links. In addition, different LLMs may have recognizable stylistic patterns, response structures, or residual boilerplate language. Therefore, residual rater unblinding remains possible despite anonymization and standardized scoring procedures.

Third, all prompts and model outputs analyzed in this study were in English, and the readability formulas used here are primarily validated for English-language text. Although Chinese guideline–derived prompts were translated and reviewed before model querying, the findings may not directly apply to Chinese-language or other non-English patient education settings. In addition, readability scores were calculated using a single online calculator, which may introduce calculator-specific variation despite consistent application across all outputs.

Fourth, the prompt set was not composed exclusively of layperson-style patient education prompts. Several prompts were derived from clinical guidelines or included specialized AMD-related terminology, including prompts related to diagnostic coding or optical coherence tomography angiography–based management of nonexudative macular neovascularization. These clinician-adjacent prompts may have induced more technical responses and contributed to the high reading-grade levels observed. Readability estimates for responses containing alphanumeric diagnostic codes or list-based ICD-10 (International Classification of Diseases, Tenth Revision) outputs may be mechanically distorted because such content can affect syllable counts, word counts, and sentence segmentation. Consequently, the findings should be interpreted as reflecting a mixed AMD information-seeking scenario rather than a pure layperson FAQ setting.

Fifth, hallucination was not prespecified as an independent analytic end point. In this study, hallucination was operationally understood as fabricated clinical facts, guideline-inconsistent recommendations, unsupported certainty regarding diagnosis or treatment, or nonverifiable references presented as evidence. Although the clinical raters evaluated reliability, transparency indicators, overall quality, and readability using DISCERN, EQIP, GQS, and JAMA criteria, they did not formally record hallucination frequency, perform claim-level fact checking, or assign binary hallucination labels to each response. Therefore, the present findings should not be interpreted as a direct estimate of hallucination risk, guideline concordance, or clinical safety. This limitation is clinically important because hallucinated AMD-related information could mislead patients about disease prognosis, urgency of anti-VEGF therapy, monitoring intervals, nutritional supplementation, or the distinction between neovascular and nonexudative disease. Future studies should incorporate prespecified hallucination detection, claim-level verification against current clinical guidelines, interrater agreement for hallucination labels, and model-level hallucination frequency reporting. In addition, the JAMA benchmark was originally developed for online health information rather than conversational AI. Because unprompted LLMs do not have conventional human authorship or financial disclosure structures, low JAMA scores should be interpreted as reflecting limited visible transparency and verifiability within model outputs rather than as direct evidence of poor factual accuracy or clinical unsafety.

Conclusions

In this comparative evaluation of 6 publicly accessible LLMs, AMD-related patient-facing responses showed substantial model-dependent differences in informational reliability, transparency indicators, overall quality, and readability. No model met the recommended sixth-grade readability benchmark under zero-shot, single-turn prompting conditions, indicating that current LLM outputs remain insufficient as directly deployable education materials for patients with AMD. These findings support the need for clinician oversight, readability optimization, and further evaluation using repeated sampling, multiturn interactions, and claim-level accuracy assessment before LLM-generated AMD information is used in patient-facing settings.